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This  thesis  develops  a  model  which  is  intended  to  be 
used  to  summarize  historical  data  pertaining  to  systems  that 
have  experienced  changeover  from  Developmental  Testing  (DT) 
to  Operational  Testing  (OT) .  Using  this  historical  data, 
maximum  likelihood  is  used  to  estimate  the  magnitude  of  the 
changeover  factor  from  the  DT  rate  to  OT  rate  and  to  predict 
the  OT  performance  of  a  new  system  which  has  undergone 
developmental  testing.  Using  a  re-sampling  method  called 
the  Bootstrap,  the  sampling  variance  and  standard  error  of 
the  changeover  factor  are  calculated,  as  are  confidence 
intervals  for  the  OT  failure  rate  of  a  new  system.  These 
estimates  and  confidence  intervals  will  provide  the 
decisionmaker  with  an  appreciation  of  the  adequacy  of  their 
projection  of  future  OT  experience  and  also  some  guidance  as 
to  the  readiness  of  his  new  system  for  entering  the 
Operational  Testing  phase. 
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EXECUTIVE  SUMMARY 


As  the  Department  of  Defense  budget  continues  to  shrink 
and  face  intense  scrutiny,  it  is  extremely  crucial  that  the 
possibility  of  savings  in  all  defense-related  activities  be 
explored,  with  the  intention  that  money  saved  be  reallocated 
profitably  to  other  areas.  Operational  Testing  (OT)  is  one 
important  activity  in  which  it  may  be  possible  to  reduce 
certain  spending  amounts  or  levels  and  thus  realize  great 
savings  for  alternative  investment. 

The  purpose  of  this  thesis  is  to  develop  a  model  which 
will  summarize  historical  data  pertaining  to  systems  that 
have  experienced  changeover  from  Developmental  Testing  (DT) 
to  Operational  Testing  (OT) .  Using  this  historical  data, 
maximum  likelihood  is  used  to  estimate  the  magnitude  of  a 
changeover  factor  from  the  DT  rate  to  OT  rate  and  to  predict 
the  OT  performance  of  a  new  system.  Using  a  re-sampling 
method  called  the  Bootstrap,  the  sampling  variance  and 
standard  error  of  the  changeover  factor  are  calculated,  as 
are  confidence  intervals  for  the  OT  failure  rate  of  a  new 
system.  These  estimates  and  confidence  intervals  will 
provide  the  decisionmaker  with  an  appreciation  of  the 
adequacy  of  their  projection  of  future  OT  experience  and 


xi 


also  some  guidance  as  to  the  readiness  of  his  new  system  for 
entering  the  Operational  Testing  phase. 

This  is  all  accomplished  in  a  spreadsheet.  After  the 
original  data  set  is  entered,  the  spreadsheet  will  run  and 
provide  the  user  with  the  OT/DT  ratio  and,  for  a  new  system, 
estimate  the  OT  failure  rate  prior  to  any  OT  testing. 


I.  INTRODUCTION 

As  the  Department  of  Defense  budget  continues  to  shrink 
and  face  intense  scrutiny,  it  is  extremely  crucial  that  the 
possibility  of  savings  in  all  defense-related  activities  be 
explored,  with  the  intention  that  money  saved  be  reallocated 
profitably  to  other  areas.  Operational  Testing  (OT)  is  one 
important  activity  in  which  it  may  be  possible  to  reduce 
certain  spending  amounts  or  levels  and  thus  realize  great 
savings  for  alternative  investment.  As  an  example,  if  a 
predictive  relationship  bstween  Developmental  Testing  (DT) 
and  OT  data  can, be  found  and  quantified,  then  this 
relationship  between  DT  and  OT  data  can  be  used  to  help 
predict  future  OT  results,  thus  decreasing  the  need  for  some 
costly,  even  wasteful,  or  at  least  possibly  premature  OT- 
level  testing.  It  is  the  purpose  of  this  thesis  to  provide 
a  methodology  for  quantifying  such  a  DT-OT  relationship  in 
the  particular  suitability  area  of  reliability.  It  is 
quite  likely  that  nearly  the  same  analytical  techniques 
described  here  can  also  be  used  to  quantify  changes  in 
effectiveness  parameters  and  MOEs  in  the  effectiveness  area 
as  well . 

Test  and  evaluation  analysts  have,  potentially  at 
least,  in  their  possession  historical  operating  (time- 
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between-f allure  or  equivalent)  data  for  both  the 
Developmental  Testing  (DT)  and  Operational  Testing  (OT) 
phases  of  various  systems.  These  systems  may  be  in 
categories,  e.g.  sensors  such  as  radars,  weapons  such  as 
missiles,  radios,  engines  or  other  mechanical  or  electrical 
devices  which  tend  to  fail  randomly  when  in  use. 

Experience  indicates  that  the  failure  rate  of  a  system 
tends  to  be  lower  late  in  the  developmental  stages  of  a 
typical  new  system  than  it  is  In  subsequent  operational 
testing,  or,  ultimately,  in  the  field.  One  reason  for  this 
is  that  the  system  is  under  the  care  of  skilled 
technologists  and  operators  during  the  development  stage, 
while  later,  in  operational  testing  and  in  the  field,  it  is 
under  the  care  of  sailors  with  less  intimate  knowledge  of 
the  system  and  its  operating  parameters  and  who  have  other 
types/pieces  of  equipment  they  must  maintain  and  repair.  It 
is  thus  quite  likely  that  OT  failure  rates  will  exceed  DT 
rates  in  practice,  and  that  it  will  be  reasonable  to  predict 
the  OT  rate  of  a  new  system  from  its  DT  rate  before  such 
testing  begins. 

There  is  believed  tO  be  a  relationship  between  the  DT 
and  OT  failure  rates  that  may  be  on  the  order  of 
approximately  one-fourth.  That  is,  a  system  in  OT  will  tend 
to  fail  approximately  four  times  as  often  in  the  same 


operational  time  as  will  the  same  system  in  final  stages  of 
DT  (i.e.  after  "reliability  growth"  is  essentially 
complete) .  If  so,  this  means  there  will  be  a  rate  shift 
associated  with  the  changeover  in  the  system's  history.  The 
problem  is  to  estimate  the  likely  magnitude  of  this  change 
for  the  new  system  as  early  as  possible  in  the  developmental 
stage,  when  there  is  very  little  data  to  support  such  an 
estimate . 

In  this  thesis  a  simple  model  is  introduced  that  can  be 
used  to  summarize  historical  data  pertaining  to  systems  that 
have  experienced  changeover  from  DT  to  OT.  Using  the 
historical  data,  maximum  likelihood  is  used  to  estimate  the 
magnitude  of  the  changeover  factor,  denoted  by  $,  from  the 
DT  rate  to  OT  rate  and  to  predict  the  OT  performance  of  the 
new  system  which  has  undergone  developmental  testing. 
Confidence  intervals,  sample  variance  and  standard  error  for 
the  ^-parameter  are  obtained  using  the  modern  computer¬ 
intensive  resampling  procedure  known  as  the  bootstraps  see 
Efron  and  Tibshirani  (1993).  This  information  can  then  be 
used  to  determine  the  initial  level  and  timing  of  an 
Operational  Testing  (OT)  activity.  For  example,  if  the 
projections  from  the  DT  data  is  for  a  higher-than-required 
OT  or  field  failure  rate,  further  steps  would  be  advisable 
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before  conducting  expensive  and  time-consuming  OT  tests. 

The  end  result  is  to  provide  an  accurate  estimate  of  the  0- 
parameter,  with  a  predetermined  confidence  interval,  to  help 
reduce  unnecessary  spending/ testing. 
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II.  COMBINING  DT  AND  OT  FAILURE  DATA 


A.  MODEL  ASSUMPTIONS 

In  this  model,  the  viewpoint  that  history  potentially 


provides  records  of  failures  both  before  and  after  a 
changeover,  e.g.  from  DT  to  OT,  is  taken.  These  records  are 
for  I  (I>1)  systems  that, 'while  not  identical,  are  judged  to 
be  comparable. 

Suppose  that  6i  is  the  prior-to-changeover  (DT) 
failure  rate  for  system  i,  and  G>i  is  the  corresponding 
post-changeover  (OT)  failure  rate,  i-  1,2,...,J.  Assume 
that  prior  to  changeover,  system  i  fails  in  accordance  with 
a  Poisson  process  with  rate  5i.  Over  operating  (exposure) 
time  Xif  system  i  fails  di  (di  =  0,1,2,...)  times  with 


probability  e~ 


.After  changeover,  that  same  system 


fails  Wi  (Wi  =  0,1,2, _ )  times  in  operating  time  yi  with 


probability  e 


It  is  assumed  that  the  data 


initially  available  are  di,  xi,  Wi,  yi  (i  =  1,2,..).  The 
objective  is  to  use  these  data  to  estimate  any  consistent 
change  in  rates  (<5i,  o)i)  from  prior-to  post-changeover,  and 
to  use  this  estimated  relationship  to  anticipate,  and 
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strengthen  estimates  of,  the  post-changeover  (OT)  rate  of  a 
new  system,  which  we  call  system  I+l . 

B.  MODEL 

Suppose  there  are  I  (I>1)  systems.  Let  Di  be  the 
random  variable  modeling  the  number  of  failures  experienced 
by  system  i  during  DT  exposure  time  Xi.  Let  Wi  be  the 
corresponding  random  variable  representing  the  number  of 
failures  experienced  by  system  i  during  OT  exposure  time  yi. 
This  model  assumes  that  {Di}  and  {Wi}  are  independent 
Poisson  random  variables  with  E[Di]  =  ^iXi  and  E[Wi]  = 


Here,  6  represents  an  unknown  constant  in  the  model  and  is 

called  the  changeover  factor. 

The  likelihood  function  for  this  model  is  seen  to  be 


L(S„  data)  =  Yl 

1  =  1 


.  d,  I  ,  w,[ 


(2.1) 


The  log  likelihood  is,  up  to  addition  of  irrelevant 
constants, 

InL^^  (<J,^,data)  = 


Z  +d,lnS  -{OS,y,)  +w,[ln  0  +  InS,]]  . 


Consequently 


-{x^+0y^)  + 


{d^+Wj^ 


(2.2) 


(2.3) 


6 


max 


f  31 

a'" 

t 

3e\} 

<  0.001 


To  find  the  Fisher  information,  useful  for  obtaining 
standard  errors  for  the  above,  second  derivatives  must  be 
evaluated: 


+  w^ 


=  -Yi 


35, dO 

3^  i.  _  i 


I  'M'; 


(2.10) 


(2.11) 


(2.12) 


3^t 


35,35  j 


=  0 


for 


(2.13) 


These  lead  to  asymptotic  expressions  for  variances  and 
standard  errors  of  the  various  quantities  estimated,  c.f. 

Cox  and  Hinkley  (1974) . 

An  important  use  of  the  estimate,  0,  is  to  project  DT 
data  for  a  new  system  into  the  post-changeover  OT  phase. 
Suppose  a  new  system,  (the  I+l^’^) ,  has  di+i  failures  during 
the  DT  test  time  of  Xm.  Suppose  we  compute  the  isolated  DT 


rate  estimate  for  the  new  system,  Then  a  natural 

point  estimate  for  the  failure  rate  during  OT  is 


(2.14) 
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Using  asymptotic  approximations  and  the  obvious 
independence,  the  estimated  standard  error  (se)  of  can 

be  computed: 


SE[o},,,\  =  JVaAeVar 


'i+\ 


+  Var 


+  Var 


^  1/ 


=  AVar\e 


1 


*;+i 


+  Var\9 


^  i  ^ 

^;+i 


*;-n 


V  ^/+i  /  V  ^z+i  y 


e\  •(2.15) 


This  in  turn  can  be  used  to  assign  approximate  standard 
errors  to  future  OT  performance,  such  as  the  probability 
that  the  future  system  will  exhibit  no/zero  failures  during 

a  test  or  mission  time 


P{W,,,  (2.16) 

The  standard  error  of  the  logarithm  of  (2.16)  can  be 
computed  from  the  inverse  of  the  information  matrix,  the 
elements  of  which  are  given  by  (2 . 10) - (2 . 13) ,  plus  the 
formula  (2.15) .  : 

C.  BOOTSTRAPPING 

A  simple  but  computer-intensive  re-sampling  or 
bootstrap  procedure,  Efron  and  Tibshirani  (1993) ,  enables 

one  to  assess  standard  errors  of  estimates  and 

m =^65 .  It  also  provides  a  useful  feel  for  the  adequacy 
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of  likelihood  as^ptotics.  The  procedure  used  here  for  a 
semi-parametric  bootstrap  is  as  follows: 

1.  Obtain  resamples  of  pre-changeover  (DT)  data  as  random 
numbers  from  the  Poisson  distribution  with  mean  |J,=di;  denote 
di  (jb) ,  for  the  b’"*'  resampled  value,  where  b  =  1,2,...,  B; 

2.  Obtain  resamples  from  post-changeover  (OT)  as  Poisson 

samples  with  mean  ;  denote  the  b*^^  resampled  bootstrap 

value  by  ^  ^  2,  .  •  .  . ,  3,  . 

3.  Using  dj(jb),  w.(jb) as  data,  apply  the  iterative  procedure 

(2.7)  -  (2.9)  to  compute  0(6)and  =  1,2,.. .,I;b  =  1,2,.. .,B. 

4.  Compute  =  =  The 

*•  J  b=\  *■  ^  fc=i  V 

bootstrap  standard  error  for  0  is  then  as  computed 

above . 

A  more  thoroughly  parametric  version  of  the  above  would 

re-sample  for  di  (jb)  from  the  .  Poisson  with  mean  S^x.f  and  for 

Wi(jb)  from  the  Poisson  with  mean  (9.5, where  9  and  5  are 
estimates  from  the  original  data.  Comparison  of  the  various 
estimates  and  their  standard  errors  is  informative.  For  an 
account  of  variations  on  the  basic  bootstrap,  see  DiCicciP 
and  Efron  (1996) .  ’ 
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D .  RELATED  WORK 

Since  the  late  nineteen-sixties,  applied  research  in 
the  military  systems  reliability  field  has  been  conducted 
and  is  currently  seriously  pursued  as  companies  search  for 
ways  to  save  money.  Two  organizations  that  have  conducted 
reliability  research  are  The  Department  of  Defense  and  the 
Boeing  Company,  but  others  have  done  so  as  well. 

In  1975,  Boeing  introduced  the  "K  factor"  device  in  an 
attempt  to  predict  failure  rates  that  are  environmentally 
affected.  In  general,  K  factors  (Logistic  Performance 
Factors)  are  numbers  that  are  used  to  adjust  Line 
Replaceable  Unit  (LRU)  field-experience  data  from  one 
environment  to  make  predictions  about  the  LRU  performance  in 
another  environment.  The  K  factor  is  a  ratio  of  the  same 
statistic  taken  from  data  sets  from  two  different 
environments  (McCabe,  Pearse  and  Rise  1975) .  It  is  quite 
analogous  to  the  factor  6  defined  earlier;  the  6  factor 
relates  specifically  to  the  OT-to-DT  failure  rate 
relationship.  Essentially,  Boeing  is  looking  at  an 
analogous  statistic,  e.g.  failure  rate,  from  two  different 
environments  and  taking  the  ratio  between  the  failure  rates 
for  systems  in  those  environments. 

The  Department  of  Defense  also  continues  to  conduct 
research  on  reliability  questions.  The  document  NAVSEA  OD 
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29304B  (1  November  1982)  is  a  comprehensive  practical  guide 
for  assessing  the  reliability  and  availability  of  the 
Strategic  Weapon  System  subsystems.  Here,  the  time  to 
failure  data  is  fit  to  a  distribution  model.  Classical 
methods  for  estimating  Hardware-  and  Software  reliability; 
including  reliability  point,  interval,  and  trend  estimation 
are  then  used,  based  upon  Binomial,  Exponential,  Normal  and 
Weibull  distributions,  to  fit  current  field  data  to  a 
specified  distribution.  Goodness  of  Fit  tests  are  then 
conducted  to  appraise  model  adequacy. 

This  thesis  provides  new  ways  to  predict  and  interpret 
data  of  similar  type.  Using  both  DT  and  OT  data,  this 
thesis  estimates  the  assumed  constant  ratio  of  the 
respective  failure  rates  (OT  to  DT) ,  for  use  in  determining 
new  OT  system  failure  times  through  sampling  over  a  whole 
range  of  possible  outcomes.  Using  equations  derived  in 
Chapter  II,  Confidence  Intervals  are  obtained  for  a  new 
system's  OT  failure  rate  from  the  system's  DT  failure  rate. 
The  methods  used  here  for  deriving  the  presumed  adjustment 
factor  (0,  or  K)  are  applicable  in  the  K-factor  setting. 

The  procedures  for  furnishing  confidence  limits  on  6 
(Fisher  information  and  bootstrapping)  are  also  applicable 
to  judge  the  uncertainty  of  K  factors,  although  the 
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reliability  literature  seems  to  have  no  published  record  of 
using  the  bootstrap  technique  for  the  present  problem. 
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Ill .  RESULTS 


The  procedures  of  Chapter  II  were  implemented  using 
EXCEL  5  and  its  accompanying  macro  language  (Visual  Basic) . 
Details  of  the  implementation  can  be  found  in  the  Appendix. 
The  implementation  given  is  the  principle  result  and  product 
of  this  thesis. 

The  implementation  of  the  algorithms  derived  in  Chapter 
II  are  now  illustrated  for  a  particular  example.  The 
hypothetical  historical  data  used  in  this  example  can  be 
found  in  Table  3.1.  It  is  inspired  by  an  example  used  by 
Anderson  (1994) .  We  assume  that  a  new  system  has 
dj+j  =  4  failures  over  the  period  Xj+i  =  666.67  in 

developmental  testing.  Unknown  to  the  analyst/  The  true 
value  of  ^  is  4  and  the  true  value  of  Sj+i  is  0.00  6.  Thus 
the  true  value  of  fi>i+j=(»2i=0 . 024 .  The  estimation  procedure 
is  applied  to  estimate  these  parameters. 


Dt 

DT 

OT 

OT 

System 

Time 

Failure 

Time 

Failure 

1 

20000 

4 

5000 

4 

2 

10000 

4 

2500 

4 

3 

6666.67 

4 

1666.67 

4 

4 

4 

4 

5 

■eh 

4 

4 

6 

4 

500 

4 

7 

4 

250 

4 

8 

4 

166.67 

4 

9 

4 

125 

4 

10 

400 

4 

100 

4 

TABLE  3.1.  Data  Set  for  10  Systems 
(Anderson. 1994,  p. 22) 
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To  illustrate  possible  results  in  practice  the  EXCEL 
spreadsheet  and  associated  macros  were  run  to  obtain  100  and 

400  bootstrap  samples:  values  of  ^b)  and  (i>)^b=l,2 . B 

for  B=  100  and  400.  Tables  3.2  and  3.3  present  results  of 
10  trials  of  the  Bootstrap  procedure.  Table  3.2  presents 
results  for  100  Bootstrap  samples;  Table  3.3  presents 
results  for  the  400  bootstrap  sample.  The  results  for  the 
400  bootstrap  cases  are  more  consistent  than  the  100 
bootstrap  cases.  It  is  recommended  that  at  least  a  400 
bootstrap  sample  be  analyzed  in  any  real  situation. 

Displayed  under  replications  of  6  are  the  point 

estimate  (bootstrap  mean  '6 )  and  the  95  percentile  bootstrap 
confidence  interval  for  6  .  The  0  Bootstrap  data  are 
ordered  from  smallest  to  largest.  By  deleting  the  top  and 
bottom  2.5  numbers  for  a  100  point  data  run,  a  95% 
confidence  interval  is  determined:  this  means  that  the 
limits  are,  respectively,  the  average  of  the  second  and 
third  smallest  for  the  lower  limit,  and  the  second  and  third 
largest  for  the  upper  limit.  For  a  400  data  point  run,  the 
top  and  bottom  10  numbers  are  deleted.  The  confidence 
limits  are  now  the  smallest  and  largest  of  the  remaining 
ordered  bootstrap  values. 
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Displayed  under  replications  of  co  are  the  bootstrap 
estimate  of  ®i+i  (the  sample  mean) ,  an  estimate  of  the 
standard  error  of  ®i+i,  and  a  95%  confidence  interval. 
Standard  Error  estimates  for  each  trial  were  obtained  by 
taking  the  square  root  of  the  variance  of  the  bootstrap 
values,  where  the  variance  is  given  by  equation  (2.14).  The 
95%  confidence  interval  is  predicted  by  listing  the  0  data 
in  ascending  order  and  deleting  the  top  and  bottom  2.5 
numbers  for  a  100  point  data  run  (top  and  bottom  10  for  a 
400  data  point  run)  as  above.  Also  shown  are  histograms  of 
the  bootstrap  distribution,  which  tend  to  be  asymmetric, 
i.e.  positively  skewed. 

Table  3.2  shows  the  niimerical  results  obtained  as 
described  above  for  each  section.  What  this  table  shows  is 
that  for  10  separate  trials,  the  spreadsheet  can  take  the 

point  estimates  for  6  and  (Oj^i  and  provide  practically  self- 
consistent  95%  confidence  intervals  for  those  values. 
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5LE  3.2  Point  Estimates  and  95%  Confidence  Intervals 

For  a  100  Data  Point  Run 

jures  3-1  and  3-2  are  histograms  of  the  100  bootstrapped  9 
i  (0  values.  These  figures  are  included  to  graphically 


Histogram 


0  Frequency 


Bin 

Figure  3-2.  Histogram  of  100 
Bootstrapped  ©'s 

Table  3.3  displays  the  results  for  a  400  bootstrap  run 
of  the  spreadsheet.  Figures  3-3  and  3-4  graphically 
display  the  positive  skewness  of  the  data  set. 


- 1 

THETA 

- 1 

OMEGA 

■■■■ 

■■■■ 

Trial 

mstasm 

SE 

Cl  Lower 

REst 

SE 

1 

2.366 

Biai 

0.016 

0.013 

0,063 

2 

5.143 

2.489 

6.222 

0.039 

0.019 

KEESI 

0.055 

3 

4.400 

2.387 

6.118 

0.033 

0.013 

0.056 

4 

0.942 

6.069 

0.028 

0.014 

0.053 

5 

0.908 

6.000 

0.033 

0.014 

0.004 

0.054 

6 

■clrAM 

0.945 

6.000 

0.017 

0.013 

0.004 

0.060 

7 

0.879 

5.935 

0.036 

0.014 

0.004 

0.054 

8 

0.973 

2.419 

6.125 

0.020 

0.014 

0.005 

0.055 

9 

KEH 

0.944 

2.531“^ 

6.250 

0.012 

0.014 

0.000 

0.057 

10 

5.333 

0.915 

2.571 

6.194 

0.040 

0.014 

0.005 

0.053 

AVG 

4.001 

0.934 

2.495 

6.107 

0.027 

0.014 

0.004 

0.056 

TABLE  3.3  Point  Estimates  and  95%  Confidence  Intervals 

For  a  400  Data  Point  Run 
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Histogram 


Bootstrapped  0's 


Bootstrapped  o's 


The  evidence  from  this  example  is  that  the  procedure 
developed  and  implemented  in  this  thesis  can  be  a  useful 
tool  for  an  operational  test  planner  and  data  analyst. 
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IV.  SUMMARY,  CONCLUSIONS  AND  RECOMMENDATIONS 

A.  SUMMARY 

The  purpose  of  this  thesis  was  to  introduce  a  simple 
model  that  can  be  used  to  summarize  historical  data 
pertaining  to  systems  that  have-  experienced  changeover  from 
Developmental  Testing  to  Operational  Testing.  Using  the 
historical  data,  the  model  estimates  the  magnitude  of  the 
changeover  factor,  denoted  by  6,  from  the  DT  rate  to  OT 
rate  by  maximum  likelihood  and  predicts  the  OT  performance 
of  the  new  system  which  has  undergone  developmental  testing. 
Throughout  this  thesis,  it  was  assumed  the  data  for  a  system 
came  from  a  Poisson  distribution,  which  is  equivalent  to 
assuming  that  times  between  successive  failures  have  the 
same  exponential  distribution  and  are  independent.  As  a 
result,  Poisson  random  variables  are  drawn  from  the  original 
data  for  each  system  using  the  numbers  of  DT  or  OT  failures 
as  the  mean.  In  doing  this,  we  can  run  the  bootstrap  and 
find  the  estimate  of  the  OT/DT  ratio.  This  is  important 
because  accurately  predicting  a  new  system' s  OT  rate  can 
save  thousands  of  dollars  in  testing  by  knowing  when  and 
when  not  to  test,  based  upon  requirements  and  confidence 
intervals. 
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B.  CONCLUSIONS 


The  analysis  and  results  indicate  that  estimating  a 
system' s  OT  failure  rate  given  that  system' s  DT  failure  rate 
and  failure  data  on  previous  similar  systems  produces  a 
reasonable  point  estimate  of  the  actual  future  OT  failure 
rate  and  confidence  intervals  on  that  rate.  With  these 
confidence  intervals,  an  operational  test  planner  is 
informed  when  an  OT  test  is  really  timely (passable) ,  or 
alternatively,  is  inconsistent  with  test  and  future  mission 
success  and  should  be  postponed. 

C.  RECOMMENDATIONS  FOR  FURTHER  STUDY 

Although  there  is  a  scarcity  of  references  concerning 
this  field  of  study,  there  are  a  few  areas  in  which  to  look 
for  future  help.  Boeing  Company  did  research  in  determining 
what  they  called  the  "K-factor"  for  failure  rates.  Although 
there  are  few  references  in  this  field,  there  is  research  to 
be  done  in  determining  the  relationship  between  this  "K- 
factor"  and  the  9  .  Another  area  of  further  study  should  be 
to  collect  actual  DT  and  OT  data  and  then  analyze  it  using 
this  thesis'  EXCEL  Spreadsheet.  This  has  been  very 
difficult  because  most  companies  are  very  "secretive"  about 
their  data  and  who  will  be  using  it.  Companies  who  are 
conducting  research  in  this  field  and  may  be  interested  in 
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sharing  their  data  are:  Boeing,  Institute  for  Defense 
Analysis,  and  AT&T  Bell  Laboratories. 
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APPENDIX:  EXCEL  SPREADSHEET  INFORMATION 

This  appendix  explains  each  worksheet  in  the  EXCEL  5 
workbook  that  comprises  the  actual  calculations  of  this 

r 

thesis.  A  sample  of  this  workbook  is  included  as  a  visual 
reference  for  the  reader. 

Sheet  one  is  entitled  "RAW  DATA".  This  is  where  the 
actual  DT/OT  data  set  is  entered  into  a  table.  This  data 
table  consists  of  the  following  columns:  System  Number,  DT 
Time  for  failures  (cumulative),  number  of  DT  failures, 
ciomulative  time  for  OT  failures,  and  number  of  OT  failures. 
The  next  sheet  is  "CONVERGENCE".  Here  an  estimate  of 

the  OT/DT  failure  rate's  ratio  [9)  is  determined  using 
equations  (2.7)  through  (2.9).  The  first  page  consists  of 
the  original  table  from  "RAW  DATA"  to  help  prevent  switching 
between  worksheets  when  locating  numbers  or  answering 
questions.  Below  this  table  a  "rough"  estimate  of  the  OT/DT 

ratio  9  is  given;  it  is  obtained  by  determining  the  DT  and 
OT  Failure  rates  and  then  dividing  the  mean  of  the  OT 
Failure  Rate  by  the  mean  of  the  DT  Failure  Rate.  To  the 
right  of  the  data  table  is  the  iterations  table.  This 
iterations  table  is  set  up  to  estimate  the  OT/DT  ratio  using 
maximum  likelihood.  Down  the  left  hand  side  is  the 

estimated  9  column  for  each  iteration  from  equation  (2.8). 
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Across  the  columns  is  SjYj  for  each  system  with  obtained 
by  equation  (2.9).  The  last  column  is  titled  SUMM  because 
here  is  where  the  denominator  of  the  right  hand  side  of 
equation  (2.8)  is  obtained.  There  are  35  iterations  in  the 

table.  The  number  "1"  was  chosen  as  the  initial  ^  value. 

Experimentation  with  various  initial  0  values  both  positive 
and  negative  resulted  in  convergence  within  the  first 
fourteen  iterations.  All  runs  of  this  spreadsheet  prior  to 

completion  of  this  thesis,  with  ^initial  =  1^  resulted  in 
convergence  within  the  first  fourteen  iterations.  One  could 

also  choose  equal  to  the  rough  estimate  described 

earlier.  This  would  result  in  fewer  iterations  if  the  rough 

estimate  was  close  to  the  true  0;  but  a  greater  number  of 
iterations  is  possible. 

Now  that  a  DT/OT  rate  relationship,  has  been 
estimated,  the  next  sheet,  entitled  "CONVERGENCE  POISSON 
DATA"  will  go  one  step  further  and  obtain  Bootstrap  samples 
of  0.  For  each  system,  a  Poisson  random  variable  is 
generated  with  mean  |j,  =  DT  number  of  failures  for  the 
original  i*^^  system  (For  the  case  of  w(b),  mean  jj,  =  OT 
number  of  failures) .  These  random  numbers  replace  the 
original  data  in  the  table  and  the  ratio  0  for  the  generated 
data  is  now  estimated  using  maximum  li)<elihood. 
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The  sheet  entitled  "100  REPS"  now  calculates  100  such 
estimates.  This  sheet  can  be  expanded  to  perform  this 
function  200,  300  or  more  times.  Using  Fisher  information 
and  step  4  of  the  semi -parametric  Bootstrap,  mean,  variance 
and  standard  error  can  be  calculated  for  the  bootstrap 
sample,  which  eventually  lead  to  calculations  for 
determining  confidence  intervals  for  ^ ,  using  normal 
theory. 

Consider  a  new  system  which  undergoes  developmental 
testing  for  a  time  Xj+i  and  which  has  a  failure  rate  5i+i.  A 
random  number  di+i  is  generated  from  a  Poisson  distribution 
having  mean  //  =  new  system's  failure  rate  ^i+i 

is  estimated  using  .  A  prediction  of  the  new  systems  OT 

■^I  +  l 

failure  rate  is  ^  ^i+i^  •  Confidence  intervals  can  be 

obtained  for  using  asymptotic  normal  theory  with 

standard  error  (2.15)  and  the  bootstrap.  For  the  b 
bootstrap  resample  of  di-t-i  r  dj+i  (b)  /  generate  a  Poisson  random 

number  having  mean  //  =  . 
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Convergence 


30 
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